NILPOTENT GELFAND PAIRS 
AND SPHERICAL TRANSFORMS OF SCHWARTZ FUNCTIONS 
II. TAYLOR EXPANSIONS ON SINGULAR SETS 



VERONIQUE FISCHER, FULVIO RICCI, OKSANA YAKIMOVA 

Abstract. This paper is a continuation of JS], in the direction of proving the conjecture 
that the spherical transform on a nilpotent Gelfand pair (A, K) establishes an isomorphism 
between the space of A-invariant Schwartz functions on N and the space of Schwartz func- 
tions restricted to the Gelfand spectrum £x>, properly embedded in a Euclidean space. 

We prove a result, of independent interest for the representation theoretical problems 
that are involved, which can be viewed as a generalised Hadamard lemma for A-invariant 
functions on N. The context is that of nilpotent Gelfand pairs satisfying Vinberg's condition. 
This means that the Lie algebra n of N (which is step 2) decomposes as D © [n, n] with V 
irreducible under K. 
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1. Outline and formulation of the problem 



We say that (N,K) is a nilpotent Gelfand pair (n.G.p. in short) if N is a connected, 
simply connected nilpotent Lie group, K is a compact group of automorphisms of N, and 
the convolution algebra L 1 (N) K of i^-invariant integrable functions on is commutative. 
This is the same as saying that (K x N, K) is a Gelfand pair or that is a commutative 
nilmanifold according to [181 Chapter 13]. 
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By B)(N) K we denote the left-invariant and i^-invariant differential operators on N, and 

by 

V=(D 1 ,...,D d ) , 

a (i-tuple of self-adjoint generators of 3(N) K . To each bounded fT-spherical function (p on 
N we associate (injectively) the <i-tuple £(ip) = \£i(<p), ■ ■ ■ , £<f(<^)) of eigenvalues of ip as an 
eigenfunction of Di, . . . , respectively. 

The (i-tuples form a closed subset of M. d which is homeomorphic to the Gelfand 
spectrum £ of L 1 (N) K ,i.e., the space of bounded spherical functions with the compact-open 
topology [6J. If (p^ is the spherical function corresponding to £ G £x>, the spherical transform 

(1) GF(£)= [ F{x)i Pi {x- 1 )dx , 

can then be viewed clS cl function on £x>. 

The following conjecture has been formulated in [7J. 

Conjecture. T7ie spherical transform maps the space S{N) K of K -invariant Schwartz func- 
tions on N isomorphically onto 

S(Z v )^S(R d )/{f:f^ = 0}. 

The inclusion Q (S(N) K ) D «5(Ex>) is known to hold in general [21 Ej, so that the conjecture 
only concerns the opposite inclusion. Moreover, the validity of the conjecture does not depend 
on the choice of T> [7]. 

In a nilpotent Gelfand pair (N,K) the group iV is at most step-two (3]. We denote by n 
its Lie algebra and by a ii'-invariant complement of the derived algebra [n, n]. We consider 
n endowed with a K- invariant scalar product. 

We refer the reader to [U [2] for the proof of the conjecture when N is either abelian or 
the Heisenberg group , and to [7J [S] when the following conditions are satisfied : 

(i) the if-orbits in [n, n] are full spheres, 

(ii) K acts irreducibly on t). 

In this paper we remove condition (i), still keeping condition (ii). The pairs for which (ii) 
holds have been classified by E. Vinberg in [T7J, and for this reason we call (ii) Vinberg's 
condition. Notice that, under Vinberg's condition, [n, n] = 3, where 3 is the centre of n. 

We mention here that the classification of nilpotent Gelfand pairs has been completed in 
P21 ED], see also [3H Chapters 13, 15]. 

We prove a preliminary result in the direction of proving the conjecture for n.G.p. satisfy- 
ing Vinberg's condition. We believe that this result is of independent interest, and its proof 
requires an interesting combination of methods from noncommutative harmonic analysis and 
invariant theory. The proof of the conjecture for pairs satisfying Vinberg's condition will 
appear in [9]. 

The proof relies very much on explicit knowledge of the pairs at hand and on the fact 
that they share some common properties. Assuming Vinberg's condition and disregarding 
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the pairs considered in [TJ El IE] , the basic list of n.G.p. to look at is that contained in 
Table [TJ Some explanations are necessary from the beginning: 

(i) in each case, the Lie bracket [ , ] : x i — > 3 is uniquely determined by the 
requirement of being A'-equivariant (see [181 Section 13. 4B] for explicit expressions); 

(ii) under the action of K, the space 3 decomposes as 30 © 3, where 3 is the (possibly 
trivial) subspace of A'-fixed elements and 30 is its A'-invariant complement. 
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Table 1. 



All other nilpotent Gelfand pairs satisfying Vinberg's condition are obtained from those 
in Table [TJ by either of the following operations: 

(a) normal extensions of K: replace K by a larger group K# of automorphisms of N 
with K<K#; 

(b) central reductions: if 3 has a nontrivial proper i^-invariant subspace s, replace n by 
n/s. 

In [8] we proved that if N, K, K# are as in (a), and the conjecture is true for (N, K), then 
it is also true for (N,K#). It will be proved in (9] that, applying a central reduction to a 
pair for which the conjecture is true, the resulting pair also satisfies the conjecture. We will 
therefore concentrate our attention on the pairs in Table [TJ 

In order to formulate our main result, Theorem |1.1| below, we need to describe some 
aspects of the structure of the Gelfand spectrum S and the way they reflect on its embedded 
copy in Mr. 

In S we distinguish a relatively open and dense "regular set" from a "singular set", and 
singular points may have different levels of singularity. Since all bounded spherical functions 
are of positive type [8], a bounded spherical function on can be expressed as an average 
over K of matrix entries of some irreducible unitary representation of N. Hence we can 
associate to each bounded spherical function a A'-orbit of characters of exp3. 

The regular elements of £ are those associated to orbits of maximal dimension. Among 
singular points, the highest level of singularity is reached by the bounded spherical functions 
associated to the characters of exp3 which are fixed by all of K, i.e., which are trivial on 
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exp3o- These are the spherical functions which factor to the quotient group N = N / exp^- 30 
We call S° the subset of E consisting of these spherical functions, and E^, the corresponding 
subset of £x>. 

At this point it is convenient to introduce a preferred system T> of generators of 3(N) K , 
obtained, via symmetrisation, from the bases of fundamental i^-invariants on n listed, case 
by case, in Section 7 of [8]. We denote by p = (pi, . . . , p<i) the <i-tuple of these polynomials. 

The polynomials pj have the property of being homogeneous in each of the variables dGD, 
z G 3o> t G 3. For each j, we denote by [j] the degree of pj in the 3 -variables. 

Notice that [j] > if and only if Dj annihilates all the spherical functions which factor 
to N. At the same time, the polynomials pj with [j] = provide a system of fundamental 
if-invariants on the Lie algebra of N, ft = © 3, where [ft, ft] = 3. Symmetrising pj on 
N produces an operator Dj G B>(N) K , which is the push-forward of Dj via the canonical 
projection. 

Suppose that the Dj G V have been ordered so that Di, . . . , D do are the operators with 
[j] = 0. Then Ejp can be realised as the intersection of with the coordinate subspace 

E° = G Z v : U+i = ■ ■ ■ = = 0} ■ 

What has been said above shows that there is a natural identification of Ejp with the 
Gelfand spectrum E^, of the pair (N, K), with D = {-Di, . . . , D do }. 

We will decompose the variables of R d as £ = (£',£"), with £' = (£ 1; . . . , £ do ), £" = 
(£do+i, • • • , £d)- To have a consistent notation, multi- indices a" will have components indexed 
from do + 1 to d, so that monomials £° only depend on £" and, similarly, 

= fl£ff ■ ■ ■ • 

We set [a"] = Yl d j=d +i\j] a j- ^f course, [a"} equals the order of derivation of D a " in the 
30- variables. 

Let us go back to the conjecture. Given a function F G S(N) K we are interested in proving 
that its spherical transform (JT1) extends from E^ to a Schwartz function on W*. In [8J, one 
of the crucial points in the proof was Proposition 5.1, providing a Taylor development of QF 
along the singular set; in that situation, there was just one level of singularity. 

Recast in our present situation, that result can be phrased as follows: given k G N, there 
exist i^-invariant Schwartz functions {-F Q »}[ a "]<fc-i on N, with QF a n only depending on 
and such that 

(2) F= D«"F a „+J2^Rp, 

[a"]<k-l \/3\=k 

with Rp G S{N) for every (3. 

It is clear, by induction, that it will be sufficient to show that the remainder term 
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can be further expanded as 

(3) ^ = E Da " F "" + E d " s y 

[a"]=k M=fc+1 

for some new functions F a n G S(N) K , [a"] = k, with only depending on and some 
new S 7 G S(N). 

Formula ^ can be seen as a noncommutative Hadamard-type formula. Its simplest 
abelian relative is the statement that if a radial Schwartz function on IR n is a sum of second- 
order derivatives of Schwartz functions, then it is the Laplacian of a radial Schwartz function 
(see also Section [2]). 

We now give the argument that allows to reduce the proof of ^ to proving Theorem 
It is convenient to introduce modified versions of the operators Dj, an operation that 
corresponds to replacing the group iV with the direct product N = N x 3 of N and the 
additive group 30 • We remark that (N, K) is also a Gelfand pair (not satisfying Vinberg's 
condition), as it can be checked from the classification in |20j or, through a direct argument, 
from the fact that the Lie algebra n is a contraction of n. 

From the same system of invariants pj used to generate the differential operators Dj on N, 
we produce, by symmetrisation on N, a system T> = {Di, . . . , Dd} of generators of H)(N) K . 
We also use the same coordinates (v, z, t) G D x 3 x 3 on N, via the exponential map exp^. 
Taking advantage of this common coordinate system for N and N, we can compare Dj and 
Dj as follows: the left-invariant vector field corresponding to the basis element e u G t) is 

X v = d Vv + ^2 b i(v)d Zi + E c i( v )9t e 



on N, and 



on N. Therefore, 



X v = d Vu + ^2ci(v)d tl 



37, 



a,A7:|j8[>l 

where each term contains at least one derivative in the variables. 

This implies that, if [a"} = k, then each term in D a — D a contains at least k + 1 
derivatives in z. Then it will be sufficient to prove ^ with each D a " replaced by D a " , since 
the difference can be absorbed in the remainder term. Therefore ^ is equivalent to 

(4) = E f)a " F -" + E d " s -y ■ 

[a"]=k |7|=fc+l 

To both sides of Q we apply Fourier transform in the z- variables, that we denote by 
e.g., 

F^,(vX,t)= / F a ,,(v,z,t)e- i{z >° dz , 



jo 
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where ( , ) is the given fT-invariant scalar product on 30- We obtain: 

(5) **(«,£*)= E DfF^,(v,(,t)+ E (<) 7 S>,C,t), 

[a"]=fc |7|=fc+l 

where each Z)j^ is obtained from .Dj by replacing each derivative d Zt by z(^. 

Modulo error terms that involve higher-order powers of (, we are left with proving that 
the k-th order term in the Taylor expansion in ( of $fc(t>, (, t), i.e., 

equals the fc-th order term in the Taylor expansion in ( of ([5]), i.e., 

£ Df #?(v,0,t) . 

[a"]=fe 

This equality is the subject of our main theorem. 
Theorem 1.1. Let G be a K -invariant function on N of the form 

(6) G(«,(,i)=^CG 7 (M) , 

G 7 G S(N). Then there are H a n G S(N) K , for [a"} = k, such that 

(7) G= b f R -" • 

[a"]=k 

More precisely, given a Schwartz norm \\ \\r p -\, the functions H a ii can be found so that, for 
some q = q{k,p), ||# a "ll(p) < C k, P Y,\j\=k \\ G -/\\(q)> f or ever V a " > \ a "\ = k - 

In Section [2j we prove Theorem LI for the pairs in the first block of Table [T] Indeed, in 
these cases the group N is reduced to t) and is abelian. 

The rest of the article will be devoted to the proof of Theorem 1.1 for the other pairs, 
where N is a Heisenberg group, with the exception of line 7, where it is a "quaternionic 
Heisenberg group" with Lie algebra HP © Im HI. 

In Section |3j we develop a careful analysis of the structure of the K- invariant polynomials 
on ©30, describing the if-invariant irreducible subspaces of the symmetric algebras over 
and 3 that are involved. 

In Section [4], we reduce the proof of Theorem LI to an equivalent problem of representing 
vector-valued ii'-equivariant functions in terms of .fT-equivariant differential operators ap- 
plied to i^-invariant scalar functions (Proposition 4.3). Then we analyse the images of these 
differential operators in the Bargmann representations of N, identifying the i^-invariant ir- 
reducible subspaces of the Fock space on which they vanish. This analysis reveals interesting 
connections between these operators and the natural action of K itself on the Fock space, 
once both are realised to be part of the metaplectic representation. 



Finally, in Sections |5j and |6| we complete the proof of Theorem LI for the pairs with TV 
nonabelian. 
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2. Proof of Theorem ll.ll for N abelian 

In this section, we consider the pairs in the first block of Table [TJ where 3o = 3 an d, 
therefore, N = is abelian. We call (t>,K) an abelian pair. In this case, one can prove 



that Theorem 1.1 is true with the additional property that the functions H a can be chosen 
independently of p and depending linearly on the G T 



Via Fourier transform in t), this statement is equivalent to the Proposition |2. 1| below. We 
first explain the notation. We split the set p of fundamental invariants into the two subsets 
p', p", where p' contains the polynomials depending only on v G D, and p" those which 
contain z G 3 at a positive power. This notation matches with the splitting of coordinates 
(£')£") 011 the Gelfand spectrum introduced in Section [lj 

Proposition 2.1. Let G G C°°{N) K satisfying 

G(v,C)= ^C 7 G» , 
| 7 |=fc 

G 7 G <S(t)). T/ien t/iere exist g a » G iS(IR do ) ; [a"] = fc ; depending linearly and continu- 
ously on {G 7 } 7 and such that 

G(v,c)= J2 p(v,(T"ga»°p'(v) . 

[a"]=k 

The proof is quite simple and relies on two adapted versions of Hadamard's Lemma on 
one side, and of the Schwarz-Mather theorem [T5l [16] on the other side. Hadamard's Lemma 
states that if a function of two variables f(x,y) G C°°(lR n x M m ) satisfies f(0,y) = for 
every y, then there exist C°°-functions gj(x, y), j — 1, . . . , n, such that 

n 

3=1 

Adapting the proof of Hadamard's lemma given in Proposition 5.3 in [8], it is easy to show 
the following. 

Lemma 2.2. Let f(x,y) G C 00 (M n x W 71 ) and fceN. Then there exist smooth functions 
g a {y) G C^iW 71 ), \a\ < k, and R a (x,y) G C°°(R n x R m ), \a\ — k + l, such that 

\a\<k |a|=fc+l 

Furthermore if f(x,y) G C 00 (1R ?1 )®5(1R" 1 ) in the sense that, for every L, 

sup (l + \y\) L \d2d$f(x,y)\ <oo , 
|a|, \(3\, \x\ < L 
yeW 1 

then the functions g a {y), \oc\ < k, and R a (x,y), \a\ = k + 1, can be chosen in iS(lR m ) and 
C °(M n )^S(W n ) respectively, and depending linearly and continuously on f . 
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Proof of Proposition 2^J_. All the polynomials pj are homogeneous in v and z and, for j = 
1, . . . , do, they only depend on v. Hence it is easy to adapt the proof of Theorem 6.1 in [2] to 
show that there exists a continuous linear operator £ : (S(v)<§)C 00 (2 l )) K -» S (R do )<§)C°° (M. d ~ d °) 
such that S(g) o p = g for every g £ («S(D)<8>C°°(3)) . So let h = £{G). Using Lemma \2.2\ 
we obtain that, for any f = (£',£") e M d , 

&(£) = E + E ^"^(0 , 

|a"|<fc |a"|=Jfe+l 

where each depends linearly and continuously on ft 6 S(JL d °)®C ao (W l ~ do ), hence on 
{G 7 } 7 . Composing with p, we get: 

o = ho P (v, o = e cr'W (//(«)) + E p( v > o a "u a » ( P (v, o) . 

|a"|<fe |a"|=fc+l 

As G is a polynomial of degree k in C, we have: 

G(v,o= E p^O^VIp'M) • □ 

[a"]=k 



3. iV NONABELIAN: STRUCTURE OF ilT- INVARIANT POLYNOMIALS ON © 3o 

From Table [l] we isolate the last two blocks, i.e., the cases where N is not abelian. To 
each line we add the list of fundamental ^-invariants on t) ©30 as it appears in Theorem 7.5 
of [H]. We split the set p of these invariants into three subsets, p , p 3o , p Vti0) containing the 
polynomials which depend, respectively, only on f £ 0, only on z £ 30, or on both v and 
z. We call the last ones the "mixed invariants". It follows from [H Corollary 7.6] that the 
algebra V(t) © 30)^ is freely generated by p = p v U p i0 U p Vt}0 . We convene to use the letters 
r,q,p to denote, respectively, elements of p„, p 30 , p D ,3 - 

The result is Table [2] Note that expressions like z k refer to the fc-th power of a matrix 
2. As in [S], at lines 9 and 10, we decompose V as the sum of two subspaces invariant under 
SpxxSpn and Spin 7 respectively: 

C 2 © C 2n = M 4n ©M 4n and C © O = M 8 ffiM 8 , 

and we write an element v of D as v = x + iy and v = Vi + if 2 accordingly. For line 10, we 
also identify M. 8 with O and the conjugation there is the octonian conjugation. 

If X is a real vector space, we call V(X) the polynomial algebra over X, and V k (X) 
the subspace of homogeneous polynomials of degree k. When X is endowed by a complex 
structure, we denote by p fcl > fc2 (X) the terms in the splitting of V(X) according to bi-degrees; 
for example V k '° is the space of holomorphic polynomials in V k . 

This applies in particular to D, which always carries a complex structure, and to 30 at lines 
4 and 5. At line 7, in fact, D admits a different complex structure for every choice of a unit 
quaternion. 
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Table 2. 



The indexing of the elements pk(v,z) of p Dj3o is assumed to match with the notation of 
Table [2] when there is more than one element in the family. 

Coherently with the notation used in the previous sections, if p a (v,z) is a monomial in 
the pk, we denote by \a\ the usual length of the multi-index a, and by [a] the degree of the 
polynomial p a {v , z) in z. When 3 is a complex space, we denote by [a] the bi-degree of p a 
in z, z. The same convention on the use of [ ] and [ ] applies to monomials in the qk- 

The pairs in Table [2] are distinguished by two properties. The first is that we can add 
a subspace 3, of dimension one or three, to 30 keeping (K, N) as a nilpotent Gelfand pair. 
The second is that D©3, regarded as a quotient of n, is either a Heisenberg Lie algebra or a 
quaternionic Heisenberg Lie algebra. Another observation will be of particular importance 
in the future. 

Remark 1. Fix ( 6 30 and let Kq be its stabiliser in K. Then the pair (N,K^) is also 
a nilpotent Gelfand pair. The result goes back to Carcano's characterisation of nilpotent 
Gelfand pairs in terms of multiplicity free actions [5j. An alternative proof can be found, 
e.g., in pZl Ch.2,§4]. 

The first dividend we get is the following. Evaluating if -invariants at £ G 30, considered 
as a point of 3, we get ^-invariant polynomials on dx{C}, or better to say on D. These 
polynomials have the same degree in v and in v [TU Section 4] . Hence the expressions of the 
polynomials r^iv) and Pk{v, z) must also have the same degree in v and v (this can be seen 
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directly from Table |2J). Therefore we have the splitting 

m,k>0 

We want to refine this decomposition, by putting special attention on the mixed invariants. 
Any mixed invariant p(v, z) in can be expanded as 



E 



3 



where, for each j, Vj and Wj are .ff- invariant, irreducible subspaces of 'P m,m (v) and V k ($o) 
respectively, with V ~ W equivalent to W as a i^-module, and 

(9) p v . jWj (v,z) = ^a h (v)b h (z) , 

h 

with {an} and {bh} being orthonormal dual bases. 

In a rather canonical way, we will now replace the basis of monomials p a (v , z)q l3 (z)r 1 (v ) 
by a new basis, obtained by replacing each p a by a new polynomial p a which is "irreducible" , 
in the sense that it equals Pv a ,w a f° r appropriate irreducible V a , W a . 

Before going into this construction, we remark some useful aspects of the list of pairs and 
invariants in Table [2j 

Remark 2. 

(a) The first block of Table [2] contains four infinite families, with both dimO and dim 30 
increasing with the parameter n. Each pair admits a single invariant in p , and 
several in p 3o and p 0ti0 . 

(b) Inside the first block, the pairs at lines 4 and 5 have a special feature, in that no is a 
complex Lie algebra and 30 is a complex space. The invariants for a pair in line 4 or 
5 can be associated with the lower degrees invariants for the pair at line 6 with the 
same D, and the former invariants coincide with those of the latter but evaluated at 
(v, —izz) instead of (v, z). 

(c) Each line in the second block contains either an "exceptional" isolated pair (line 10), 
or an infinite family (lines 8, 9), but with 30 fixed. Each pair admits a single invariant 
in p l0 and in p 0t}0 , but several in p . 

(d) For each pair, the k-th mixed polynomial Pk{v, z) is a finite sum 

(10) Pk(v,z) = ^tj(v)b jk (z) , 

j'=i 

with v\ = dim3 and the £j independent of k. 

(e) For the pairs at lines 6-10, the polynomials bji(z) appearing in the expression (10) 
of pi are the coordinate functions on 30. The real span of the polynomials ij(v) is a 
ii'-invariant subspace of P 1 ' 1 (o) equivalent to 30. 

(f) At lines 6, 7 and for k > 1, Pk(v, z) (resp. qk(z)) equals, up to a power of i, pi(v, z k ) 
(resp. qi(z k )). Here again z k is the k-th. power of a matrix. 
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Because of Remark [2] (b), we first restrict our attention to the pairs of lines 6-10. 

For given m, k, we look at the structure of (P m - m (» <g> V k {} )) , the space of i^-invariant 
polynomials on d © 30 of bi-degree (m, m) in v and degree k in z. 

Inside V m ' m {t>) consider the subspace generated by polynomials which are divisible by 
elements of p , and let 1-l m,m (v) its orthogonal complement. More explicitly, if r 7 (t>) is a 
monomial in the rj of bi-degree (5 7 , 5 7 ), then 



With an abuse of language, we call W n,m ('o) the harmonic subspace of V m,m {x>). By the 
fT-invariance of each 'H rn,m (t>), 



Similarly, we set 



A" 



w fc (3o) = ( £ ^ fc - [/3] (3 ; 



l<[/8]<fc 

For an element p of ("P m ' m (t)) ® ^(jo)) we denote by p its b-harmonic component, i.e., 

its component in (T-L m ' m (v>) ®V k (^o)) K . 

Finally, we denote by V m (£) C P m ' m (t>) the space generated by the monomials of degree 
m in the £j. 



Proposition 3.1. Let K, ti, 30 be as in Table^ lines 6-10. 

(i) If k < m, (n m ' m (v)®V k (zo)) K is trivial. 

(ii) For k = m, {H m ' m (v) <g> "P m (3o)J 

is one- dimensional, and it is generated by p™ . 

(iii) Let V m = H m ' m (v) r\V m (£) . Then V m is absolutely irreducible, i.e., it stays irreducible 
as a representation of K c after the complexification V m ®^C We fix an orthonormal 
basis a^, 1 < j < v m , of V m . Then 

Vm 

(11) PT = J2a^(v)b^(z) , 

3=1 

with the b^ non-trivial. 

Let W m denote the linear span of the b^™\ 1 < j < v m . Then W m ~ V m and 

w m c n m ( d0 ) . 

(iv) If \a\ = m, then p a 7^ and 

Vm 

3=1 



12 



(v) For every m and k, the products p a q^ with \a\ — m and [a] + [(3] = k form a basis of 
(H m ' m (o) <g> V k ($ )) K ■ In particular, 

(H m ' m (t>) <g> V k (t )) K = (V rn ® V k (zo)) K ■ 

(vi) The spaces V m are mutually K -inequivalent. 

Proof, (i) is a consequence of the structure of the pj. If k < m, a monomial in the p,q,r 
must necessarily contain some r-factor. 

(ii) follows from the fact that is the only monomial in (V m ' m (\j) ®P m (3o)) which does 
not contain r-factors. If we had p a = 0, this would establish an algebraic relation among the 
fundamental invariants, in contrast with [HI Corollary 7.6]. This last remark also proves (v) 
and the first statement in (iv). 

The proof of (iii) requires some discussion of V m (£). First of all, every element of 

(H m > m {t>)®V k {to)) K necessarily belongs to the smaller space ((H m ' m (v)nV m (£)) ®V k (z )) K , 
by the structure of the invariants. 

The second fact is that the equivariant map of Remark [2] (e), from 30 to the span of the 
£j, induces a surjective equivariant map from P m (3o) to V m (£). 

Consider now % m,m (t)) r\V m (£). For every irreducible A'-invariant subspace V of it, there 
must be an equivalent irreducible subspace W in "P m (jo)- This gives rise to an invariant 
Pv,w of Q, belonging to (V <8> W) K C (H m ' m (fc) <g) V m {^ )) K . But by (ii), this space is 
one-dimensional. Therefore there exists a unique V C r H m,m (t)) r\V m (£) and a corresponding 
unique W C P m (j ) equivalent to V. This forces V to be all of "H m ' m (» n V m (£), and it 
must coincide with V m . The equality dim (V <8> V) K = 1 implies also that V is absolutely 
irreducible. 

Decompose now p™ as 

[/3]>0 

with p^ e V m £g> 7/ m (3o) and p» 6 V m ® % m ~' /3 ](3o). Then p tt and the p^'s are all if- invariant. 
It follows from (i) that p^ = for every (3, i.e., = p" e Kw ® "H m (3o)- 

To complete the proof of (iv), take any element p of (7/ m ' m (t)) ® "P^o))^- By ([8]), 

with the Pv 3 ,Wj as in ([9]). Repeating the same argument used above, each Vj gives rise to an 

invariant polynomial in (H m ' m (t>) <g> V m (} )) K . By (iii), V,- = H m ' m (fc) n P m (f) for every j. 
We prove (vi) by contradiction. If we had V m ~ V^/ with m < m', the polynomial 
• aj m - ) (t>)6^ m ^(2;) would be a non-zero element of ("H m ' ,m '(t)) <8> 'P m '(3o)) , contradicting 

(i). 1 □ 

Consider now the pairs of lines 4, 5. Introducing bi-degrees for polynomials on 30, we 
obtain the following rather obvious variants, on the basis of Remark [2] (b). 

Proposition 3.2. Let K, D, 30 be as in Table\^ lines 4, 5. 
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(i') If k < m, (H m - m (D) ® V k ' k (to)) K is trivial. 

(ii') The polynomials p a , p a coincide with those of line 6, evaluated at (v,—izz). In 



particular, (ii), (Hi), (iv), (v) ; (vi) of Proposition 3.1 have the same formulation (up 
to the obvious notational changes), with the same and V m as for the twin pair 
of line 6. 

(in') For k > m, (n m ' m (t)) <g> V k ' k (} )) K = span{pV : \a\ = m , [a] + = (k, k)}. In 
particular, 



(H m '"» ® V k > k (l )) K = (V m <g> P fc ' fc (3o)) A ' 



(iv') // fci 7^ /c 2 , ("H m,m (o) g) P fcl ' fe2 (3o)) ^ trivial, except at line 5, for k\ — k 2 = jn, 
j G Id. In this case, 

{n M ® v (3o)) -\(m- j (n^(t>)®v^(so)) K ifj<o. 



Notice that Propositions 34_ and |3.2| show that, for every a, 

(12) P^=Pv m ,W a , 

with m = \a\ and W a C P [a] (3o) (resp. W Q C P w (3o)) equivalent to V w 



Corollary 3.3. The polynomials p a q^r 1 form a basis ofV(t> ©3 







4. Fourier analysis of JCequivariant functions on iV 



We start from a function G as in Theorem 1.1 



G(v, (,t)=J2 CG^(v, t) (lines 6-10) , 

h\=k 

G(v, C,t)= Yl C 71 C 72 G>, t) (lines 4, 5) , 

I7i| + l72|=fc 

which is iCinvariant, and with G 7 G S(N) (we use the variable ( as a reminder that, in the 
course of the argument, we have taken a Fourier transform in z). 



The following statement follows from Proposition |2.1| and Corollary |3.3 
Lemma 4.1. 

(i) (lines 6-10) A function G G (S(N) ®V k (lo)) K can be uniquely decomposed as 
(13) G(v } (,t)= Q P (0p a (v,()g a p(v,t) , 

with g a/ 3 G S(N) K depending continuously on G. 
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(ii) (lines 4, 5) A function G G (<S(N) ®V k ' k ($o)) can be uniquely decomposed as 
(14) G(v, C, t) = AOfi(v, Q 9aP {v, t) , 

M + [/3] = (fc,fc) 

with g a/ 3 G S(N) K depending continuously on G. 

(iii) For the pair at line 5, (S(N) ® V k+jn > k (z )) K equals (Pf z) j (S(N) ®P fc ' fc (3o))^ for 
j > 0, (PTz)-J(S(N)®V k > k ( d0 )) K for j < 0. 



From the right-hand side of (13), or of (1141), we extract the single term 



pty.OfttfM) = E a f ) (v)g aP (v,t)bf t) (C) , 

with m = | a | > 1. 

In order to emphasise that the following analysis depends only on m and not on the 
specific multi-index a, it is convenient to introduce an abstract representation space V m of 
K, equivalent to V m , and denote by {ej}i<j<v m an orthonormal basis corresponding to the 

basis {aj } of V m via an intertwining operator. 
We denote by r m the representation of K on V m . 
We regard the function p a g a/ 3 as a V m - valued function on N: 

Vm 

G a f)(v, t) = g a f)(v, t) ^ a< T } ( v ) e 5 m) • 



Since the form an orthonormal basis of the space W a in (12) and W a ~ V m ~ V m , it 
follows that G a/ 3 is if-equivariant, i.e., 

G a p(kv,t) = T m (k)G a p(v,t) , (k e K) . 

In fact, we have the following characterisation of i^-equivariant V m - valued smooth func- 
tions. 

Lemma 4.2. Lei H be a V m -valued, K-equivariant Schwartz function on N. Then H can 
be expressed as 

Vm 

H(v,t) = h(v,t)J2^\v)e ( f l) , 
loit/i ft G S(N) K , depending continuously on H. 

Proof. Reversing the argument above, from a i^-equivariant function H(v, t) = Hj(v, t)e^ 
we can construct the ^-invariant scalar- valued function H(v, £, i) = ^ . which 
satisfies the hypotheses of Proposition |2.l[ Hence H can be expressed as 

m'<m [o] + [/3]=m 
|«|=m' 
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with each h Q p G S(N) K . Each term in parenthesis can be turned into a f^-equivariant 
function with values in V m >. Since the V m > are mutually inequivalent, the only non-zero term 
is the one with m! — m. □ 



Remark 3. From this point on, we may completely disregard the special cases of lines 4 and 
5, because in this abstract setting they are completely absorbed by those of line 6. 

We denote by G D(iV) the differential operators obtained from the polynomials 
by symmetrisation. Then 

(15) M m = Y,e { r ] Af ] 

is a i^-equivariant differential operator mapping scalar valued functions on N to V m -valued 
functions. 

The following statement is the key step in the proof of Theorem 

Proposition 4.3. Let G be a V m -valued, K-equivariant Schwartz function on N. Then G 
can be expressed as 

(16) G(v,t) = M m h(v,t) , 
with h G S(N) K . 

More precisely, given a Schwartz norm || \\^, the function h can be found so that, for 
some q = q(m,p), \\h\\ {p) < C mjP \\G\\( q y 

The proof requires some representation theoretic considerations that will be developed in 
the next subsections. 

4.1. The Bargmann representations of N. 



The proof requires Fourier analysis on N. As we mentioned already, N is either a Heisen- 
berg group or (line 7) its quaternionic analogue, with a 3-dimensional centre. It will suffice 
to restrict attention to the infinite-dimensional representations. 

When N is a Heisenberg group, i.e., n = D © 1R, we see from Table [l] that t) is a complex 
space (whose dimension we denote by k), with K acting on it by unitary transformations. 
We use the Bargmann-Fock model of its representations, that we briefly describe. 

If (t>i, . . . ,v K ) are linear complex coordinates on t), the 2ft left-invariant vector fields 

(17) Zj = d Vj - -Vjd t , Zj = 8%. + -Vjdt , j = l,...,K, 

generate n c . 

For A > 0, the Bargmann representation tt\ acts on the Fock space ^(tj), defined as the 
space of holomorphic functions (p on t) such that 

II^H^ = (A/2tt) k / \(p(v)\*e-*W a dv < oo , 

J V 



(19) d-K X {Zj) = - Vj , dnxiZj) = d Vj 
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and is such that 

(18) dnx(Zj) = d Vj , dnx(Zj) = ~Vj . 

For A < 0, 7Ta acts on as ir\(v,t) = n\\\(v,—t), so that the roles of Zj and Zj are 
interchanged: 

A 
2 

By the Stone-von Neumann theorem, the Bargmann representations rr x , A ^ 0, cover the 
whole dual object N up to a set of Plancherel measure zero. 

The case n = t)©ImH, with t) = HP, requires some modifications. For every fi ^ in Im H, 
with polar decomposition /j, = Xq, X = \fi\ > 0, there is an analogous representations 7r^ = 7T\ )S 
which factors to the quotient algebra t\ = D ? © (Imi/^ 1 ). This is a Heisenberg algebra, 
with t) ? denoting D endowed with the complex structure induced by the unit quaternion q. 
Then tc\ s is the Bargmann representation of index A of ti f , acting on the Fock space .F(t) ? ). 

Again, the ir^ cover N up to a set of Plancherel measure zero. 

For the sake of a unified discussion, we drop the subscripts A or /i, and simply write n 
and J 7 . Only when strictly necessary, we will reintroduce a parameter A > 0, leaving to the 
reader the obvious modifications for the other cases. 

In all cases, the fact that K acts trivially on | implies that each representation as above 
is stabilised by K. In fact, if a denotes the representation of U K on functions on t) given by 

(20) (<j(k)<p)(v)=<p{k- 1 v) , 
one has the identity 

ir(kv,i) = a(k)7i(v,t)cr( y k^ 1 ) . 

The representation n maps functions H G S(N) ®V m into operators tt(H) G C^J 7 ) ®V m = 
C(J- \T ® V m ), depending linearly on H and such that 

tt(/i <g) w ) = n(h) (g) w , heS(N). 

If H is K-equivariant, then 

(21) n(H)a(k) = (a®T m )(k)ir(H) , 

for all k E K. Similarly, the equivariance of M m implies that, for k G K, 

(22) d7i(M m )a(k) = (a <g> r m ) (A;)rfvr(M m ) , 

i.e., 7r(if) and dn(M m ) intertwine a with a © r m . 

With an abuse of notation, we denote the restriction of a to K by the same symbol. 

Since (N,K) is a n.G.p., the representation a decomposes into irreducibles without mul- 
tiplicities. We can write 

(23) T = Y?V(n) , 
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for some set X of dominant weights ft of K. For each ft, we denote by Rift) the representation 
of K with highest weight ft. Each V(ft) is contained in some P s '°(t)) with s = s(ft), since 
these subspaces are obviously invariant under cr. 

In particular, V(ft) consists of C°°-vectors for 7r, so that d7i(M m ) is well defined on V(ft). 

Notice that, for the pairs in the first block of Table [2j each V s '°(ti) is itself irreducible. 
Only for the pairs in the second block, different V(/x)'s may be contained in the same V s,0 (v). 



The following lemma in invariant theory will be important in the next proof. 

Lemma 4.4. Let R(fti), R(ft2), Rifts) be three irreducible finite dimensional representations 
of a complex group G on spaces V\, V 2 , V 3 respectively. Denote by c^iftjiftk) the multiplicity 
of Rifti) in Riftj) <S> Riftk)- Then 

c m iftj, ftk) = dim (V- <g> Vj ® V k ) G = c^, (ft k , /x-) , 

where ft' stands for the highest weight of the dual representation and V for the dual vector 
space of V. Over K the statement modifies as follows: 

dim (Vi <g> Vlfc^ (ftj, ft k ) = dim (V[ ® Vj <g> V k f = c M j (/x fe , /x^dim ® \//) G • 

Proof. Recall that by a straightforward consequence of Schur's lemma, for irreducible com- 
plex representations, we have dim (V-®Vj) G = if Vi ^ V,- and dim (V7<S>V,-) G = 1 if Vi = Vj. 
Next, c^Xftjiftk) counts how many times Vi appears in Vj®V k and c^/x^, ft'i) how many 
times Vj appears in V[®V k . Finally, over the real numbers the dimension of (V-®Vj) G can 
be larger than one. □ 

Proposition 4.5. Let $ be a linear operator, defined on the algebraic sum of the V(ft), 
ft G X, with values in T <S> V m , and intertwining a with a ®r m . Then 

(i) for every ft, 

$ : V(ft) — ► V(ft) <g) V m ; 

(ii) $| vw = ; unless R(ft) C Rift) <g> r m ; 

( ni ) $ Iv ( m) =° */ s (^) < m - 

Proof. Let P M be the orthogonal projection of T onto V(ft). If /xi G X, (P Ml <8> Id)$| v , . 
intertwines Rift) with R(fti) <S> r m . Hence (P Ml £g> Id)<J>| v( . = unless P(/x) C R(fti) ®T m . 

Take W m , the linear span of the polynomials b^\z) in (11), as a concrete realisation of 
r m . Take also V(fti) as a concrete realisation of R(fti) and V(ft) as concrete realisation of 



the (complex) contragredient representation Rift)' of Rift). By Lemma 4.4 



(24) R{ft) C P(/X!) <g> r m ^ (Hu)<g>V(/ii)®W m )* ^ {0} . 

By Remark [l for a nonzero element £ G 30? the pair (N,K^) is also a nilpotent Gelfand 
pair, so that .F(t)) decomposes without multiplicities under the action of K^. Let p(t> , z) 

be a nonzero element of iV(ft)®V(fti)®W m ) , and fix ( G jo such that po(v) = p(v,() is 
not identically zero. Then p is ^-invariant and contained in V(ft)®V(fti). Hence V(ft) 
and V(fti) must contain two fQ-invariant, irreducible, equivalent subspaces. By multiplicity 
freeness, this forces that ft — fti and we obtain (i). 
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At this point, (ii) is obvious. 

To verify (iii), observe that the subspaces V m are mutually inequivalent by Proposi- 
vi), |3.2|(ii'). Hence V m does not appear in V s ' s (to) for s < m. □ 



tions 3.1 



4.2. Multiplicity of R(fi) in R(fi) ® V m . 

We need at this point to obtain, for any m, 

(a) a precise description of the "m-admissible" weights /x, i.e., such that R(fi) C R(p) <S> 

(b) that, for such a pair, R(fi) is contained in R(fi) <8> V m without multiplicities. 

Point (a) above forces us to go into a case by case analysis, from which we will obtain sets 
of parameters for the m-admissible weights. This analysis will also give us a positive answer 
to point (b). 

For a simple complex (or compact) group, we let denote its fundamental dominant 
weights. 

4.2.1. Pairs in the first block of Table\^ 

In these cases we know that V(p) = V s,0 (to) for some s. 

Proposition 4.6. Let to = C n , with K = (S)U n , or to = C 2n with K = Sp n . Then V s '°(v) is 
contained in V s '°(t>) <8> V m if and only if s > m, and in this case with multiplicity one. 



Proof. We know from Propositions 3.1 (i) and 3.2 (i'), that V s '°(v) is not contained in 
V s ' (to) <S> V m if s < m. We suppose now that s > m and apply the equivalence (24). Since 
the only fundamental invariant depending only on v is \v\ 2 , there is exactly one invariant 
(up to scalars) in V s '°(to) ® P°' s (to) ® W m , namely \ v \ 2 ( s ~ m )pf . 



By Lemma 4.4, this gives existence and uniqueness of a subspace of V s,0 (to )®V m equivalent 



to V s '°(to). □ 



4.2.2. Pair of line 8. 

The action of SU n on the C n -factor in to extends to the action of SL n (C). Depending on 
the ordering of simple roots, this latter action may have the highest weight either w\ or 
vj n -\. For convenience we assume that this highest weight is w n -\. Also let S l denote the 
representation of SU2 on V l '°(C 2 ) and by \ s the s-th power of the identity character on Ui. 
Then, cf. [12], 

i+2j=s 

We call R St i (with < i < s, s — i 6 2N) the i-th summand above, and V s ^ the corre- 
sponding subspace of V s ' (to). 

Proposition 4.7. R s ^ is contained in V Sj i <8> V m if and only if i > m, and in this case with 
multiplicity one. 
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Proof. Notice that both SU n and the centre of U2 act trivially on 30 and that the remaining 
factor SU2 of K acts on W m by S 2m . Then we want to find when it is true that R S) i C 
R S}i © S 2m . We have 

R s . g, S 2m = R(iw! + jw 2 ) © {S { © S 2m ) <8) x s 
<2 ' ,! = R{iw 1 +jw 2 ) © (S* +2m © ,S i+2m - 2 © • • • © S |2m - J| ) © x s • 

It is quite clear that we find the summand S l in the sum in parentheses if and only if 
i > |2m — i|, i.e., i > m, and in this case it appears once and only once. □ 



4.2.3. Pair of line 9. 

With the same notation of the previous case, we have, cf. [T2] 

i+2j<s i+2j<s 

s-ie2N s-iS2N 



i,3 ■ 



Proposition 4.8. R B ,%j is contained in K,«j©V m if and only ifi>m, and in this case with 
multiplicity one. 

Proof. As before, we want to find when it is true that R s ,ij C R s ,i,j © S 2m . The same identity 
(25) as above holds and we obtain the same conclusion. □ 



4.2.4. Pair of line 10. 

We can identify t> with C 8 , with Spin 7 acting via the spin representation and Ui by scalar 
multiplication. 

The spin representation defines an embedding of Spin 7 into SOg. Under the action of 
Ui x SOs, V S '°(C 8 ) decomposes into irreducibles as 

V sfi (C 8 )= J2 nivy-'W^ J2 V :*> n(v) 2 = vl + --- + vj, 

i>0 i>0 
s-iS2N s-i£2N 

see e.g. [TDJ §19.5]. 

The compact groups Spin 7 and SO§ have the same invariants on C 8 , see e.g. JSJ Theo- 
rem 7.5(8)]. Following the same line of arguments as was used in Section |3j we can conclude 
that the above decomposition is also irreducible under the action of Spin 7 . 

Therefore 

a i^,o (D) = R ( iws ) ® x ° = J2 R -^ ■ 

i>0 2i<s 
s-ie2N 

Proposition 4.9. R s ^ is contained in V Sj i © V m if and only if i > m, and in this case with 
multiplicity one. 
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Proof. The group Spin 7 acts on 30 via R(rui) (and Ui acts trivially). The orthogonal pro- 
jection of W m on the highest component R(mw\) of V m {io) must be non-zero, otherwise 



V m C V m ~ 2 {lo) and we would have an invariant contradicting Proposition 3.1 i). Therefore, 
Spin 7 acts on V m via R{mwi). 

We follow [TU Example 5.2]: setting k = m — s + i, R{mwi) ® R(kzu 3 ) decomposes as a 
direct sum 

R{mwi) <S> R(kw 3 ) = R{aiWi + a 2 tu 2 + (a 3 + a 4 )o7 3 ) , 

Il,«2,«3, a 4 

extended over the quadruples (a>j)i<j<4 of nonnegative integers such that 
(26) ai(l, 0) + a 2 (l, 2) + a 3 (0, 1) + a 4 (l, 1) = (m, k) . 



We are interested in the solutions of (26) which satisfy the requirement a\ = a 2 = and 
03 + 04 = k. It is clear that there is one (and only one) solution if and only if m < k, with 
a 3 = m, a 4 = k — m. □ 



4.3. Nonvanishing of dir(M m ) on m-admissible weight spaces. 

We have shown that, if /i is m-admissible, there is a unique subspace X(/i, m) C V{fi)®V m 
equivalent to V(/i). Therefore, Proposition 4.5 (i) can be made more precise by saying that 
an operator <3> intertwining a with a <g> r m maps V(/x) into X(/i, m) for any m-admissible \x. 
Moreover, $| V(M) is uniquely determined up to a scalar factor. 

Assume that the identity (16) holds. Applying tt to both sides, we obtain 

tt(G) = dvr(M m )7r(/i) . 



In this identity, 7r(G) and dn(M m ) satisfy the assumptions of Proposition 4.5, whereas 
7r(/i) maps each V{^i) into itself by scalar multiplication (this is the special case m = of 
Proposition 4.5). 

The next proposition, whose proof is postponed to the end of this section, provides a 
necessary condition for being able to solve equation (16) in h. 

Proposition 4.10. For every m-admissible weight \x, d7r(M m )\ v ,. ^ 0. 

Let C = (cjk) be a k x k hermitian matrix (with k = dimcD), and set 

The symmetrisation process transforms ic into the operator Lc € H>(N), 

Lc = g C ik{ZjZk + ZkZj) , 



where the Zj , Zj are the vector fields in (17). 

The image of Lc in the Bargmann representations can be described in terms of the rep- 
resentation a in (20). 
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Lemma 4.11. Let C = (cjjt) be a k x k hermitian matrix (so that iC G u K ), and let 

l c = IJ2 c ik(Z 3 Z k + Z k Z 3 ) G B(N) . 



i.k 



Then, for A > 0, 



dn x (iL c ) = ^da(iC) 



This identity extends by C-linearity to C G 5l K , understanding L c as \L c +c* — \^%{c-c*)- 

For the proof, that we skip, it suffices to verify the identity for C = E ik + E ki and 
C = iEik — iE k i- Notice that a is the restriction to U K of the metaplectic representation. 

Denote by Lj the symmetrisation on N of the polynomials £j(v) appearing in the expression 



(10) of the mixed invariants pp.- We want to identify how G?7r(span {£•/}) sits inside da(u K ) 



and understand the action on V(/x) of the complex algebra generated by the dir(Lj). By 



Lemma 4.11, this is equivalent to identifying 

c = [iC : L c G span {Lj}} 
inside u K and study the algebra generated by da(c c ). 

Proposition 4.12. As a representation space of K, c ~ Vi ~ $o- Moreover, 

(i) When 30 — su r (line 6 with r = n, or lines 8, 9 with r = 2), K contains a factor 
Kq = SU r acting nontrivially on 30 ■ Then c = t . 

(ii) For line 7, c is the Sp n -invariant complement of sp 2n in su 2n - 

(iii) For line 10, let 1 be the inclusion of Spin 7 in SOg given by the spin representation 
-R(tz7 3 ). Then c is the 7-dimensional Spin 7 -invariant complement of di{so-j) in %. 

Proof. The first statement follows from the equivalence span{L 3 } ~ span{£j} ~ Vi- 

After Lemma 
(ii), it is basical 



4.11, (i) is almost tautological: the symmetrisation ofpi(-,z) is L_ ida ^ z y For 
y the same argument. 

For (iii), we must recall from [S] that the terms V\,v% in the expression of px(v,z) = 
Re (2(^1^2)) are octonions representing the two components of v — 1 <S> i>i + i <S> V2 in the 
decomposition of C <g> O as the direct sum of K ® O and (z'M) <S> O. 

For fixed z, pi(-, z) is a quadratic form satisfying pi(v, z) = —px(v, z) (here v — 1 <S> V\ — 
i g) vz). In complex coordinates, it is then expressed by a hermitian matrix C z with purely 
imaginary coefficients. It follows that iC z G so$, and these elements span a Spin 7 -invariant 
7-dimensional subspace. This is necessarily the complement of di{soi). □ 

Notice that either c C t is already a Lie algebra, or t © c C u B is itself a Lie algebra. Set 
q := { + c. In two case, lines 7 and 9, q 7^ C, when $j is either su 2n or q = so 8 ©M, respectively. 
Let G be the corresponding compact group with g = Lie G. Also notice that if g 7^ t, then, 
up to the summand 1R, I © c is the Cartan decomposition of the symmetric pair (g, t). 



Lemma 4.13. The subspaces V(fi) in (23) are also G -invariant 



Proof. The action of K on c is equivalent to the action of K on 30 ■ Therefore for each 
iC G c, the action of the stabiliser K^c on J 7 is multiplicity free and iC preserves each of 
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the irreducible summands. Since K^c C K, the action of iC also preserves i^-invariant 
irreducible subspaces in J 7 . □ 



The statement of Lemma [4. 13| can also be verified directly using the fact that K and G have 
the same invariants on D. 

We can now prove Proposition |4.10 



Proof of Proposition J^AQ First of all, recall that we do not treat lines 4 and 5, because they 
are completely covered by line 6. 

Fix a complex basis {ui, . . . u ui } of 3q with ui being a lowest weight vector (of weight, say, 



—a) and let (zi, 



denote coordinates in this basis. Then a is also the highest weight 



of c , z™ is a vector of the highest weight, ma, in "P m (3o), and the weights ±ma do not 
appear in lower degree polynomials on 30- Hence ±ma are not among the weights of V s (£) 
with s < m. Decomposing pi(v,z) with respect to Zj, one gets 

Pi(v,z) = ^2aj(v)zj , 
i=i 

where a™ is a lowest weight vector in V m (£). We must have a" 1 G "H m,m (t)), since other- 
wise, by Corollary 3.3, the weight —ma would also be contained in lower degrees in i. By 



Proposition 3.1, the .fT-invariant space generated by a™ is V m . In turn, this implies that z™ 
belongs to the space W m of Proposition 3.1[iii). 
We regard Mi in ( 15 ) as 

i=i 



"3 J 



identifying 30 with V\. Then 
(27) 



E 



where each Bp is an m-fold composition of the Ay 

Each Bp is the symmetrisation of a polynomial 6^ depending on v and t G 3. The polyno- 
mial 

P(v,z,t) = Y,bp(v,t)z? 



is if-invariant, and its component of highest degree in v is p™ . Therefore, p™ is the highest 
weight term in the decomposition ^ of P. 

In particular, Mf 1 and M m have the same highest weight component. Then 

(M m ,z?) = (M™,z?) = A™ . 



Let X be the lowest weight element in c such that A\ = Lx- Lemma 4.11 implies that 

d7T X (A™ ) = {X/2) m da{X) m . 

Therefore it remains to show that under the identification 30 = c, the element da(X) r ' 
does not vanish on V(n). 
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As an illustration, consider first the example of line 6. Here c = to and X is a lowest root 
vector in sl n . The complex group SL n (C) acts on V(/i) via R(svj\) with s > m. Clearly 
da(X m ) is non-zero on the highest weight vector of V (/•*)• 

In general, we argue in the following way. The action of X on polynomials on t) is 
completely determined by the action of X on itself or by the representations of the group G. 
If V(p) is m-admissible and da(X m ) is zero on V(p), then it is also zero on the contragredient 
space V(/x), and, hence, da(X 2m ) vanishes on a copy of V m sitting inside V (p,)<S>V \p) C V s,s . 

Now V m has the highest weight ma and X is of weight —a. Since X is a weight vector 
(with a nonzero weight) of a torus in $j c , it is necessary a nilpotent element. Therefore 
one can include it into an s^-triple {X, H, Y } C $j c , where the semisimple element H is 
contained in t € . (If t = $j this is Jacobson-Morozov theorem, in the two cases with q ^ t the 
claim follows from the fact that (g,t) is a symmetric pair, see [T3], Prop. 4].) 

Then H multiplies a highest weight vector v G V m by 2m, therefore w gives rise to at least 
one irreducible representation of {X, H, Y} of dimension at least (2m+l). By the linear 
algebra considerations, da(X 2m )v ^ 0. A more careful analysis can show that da(Y)v = 
and da(X 2m )v is a lowest weight vector of V m . □ 



5. Proof of Proposition 14.31 



First, let us fix some notation. Let T = d t be the central derivative of N when N is the 
Heisenberg group. For the pair at line 7, where N is the quaternionic Heisenberg group, we 
take Tj = d tj , j = 1, 2, 3, the derivatives in three orthogonal coordinates on 3. 

We can assume that D = (L>i, . . . , Dd -i, i~ l T) and D = (Di, . . . , Dd -3, 2 _1 T 2 , i _1 T 3 ) 

respectively. The first do — 1 (resp. c?o — 3) operators come from symmetrisation of the poly- 
nomials pj G p D . We convene that D\ is the sublaplacian, i.e., the symmetrisation of |v| 2 . 
A point of the spectrum E^, of (N, K) can then be written as £' = (^, A) with A in R or 
IR 3 , depending on the pair considered. The points of the spectrum with A 7^ form a dense 
subset of E/j and they are parametrised by A and /i 6 las £'(A,/i), where ^(A,/i) is the 
scalar such that 

(28) d7r A (5 i )| VM =^(A l /*)Id. 

Note that Uii^v) = x and ) if v (v) c ^ s '°(f), then ^(A,/i) = |A|(2s + re), cf., e.g., [I]. 
By 8j we denote the degree of homogeneity of the polynomial pj (and hence of Dj) with 
respect to the automorphic dilations 

(29) r • (v, t) = (r^v, rt) 

of n (and of N); i.e., 8j = \degpj for the first do — 1 (resp. do — 3) operators, and 8j = 1 for 
the T's. 

If ip(v,t) is a spherical function, then ip r (v,t) = ip(r*v,rt) is also spherical, and £'j(<p r ) — 
r s i£j((p). Then is invariant under the following dilations of M. d °: 

(30) r-(^,...,4) = (r^,...,r^4) . 



24 



In terms of the parameters (A,//), we have 

r-e'(A,^)=e , (r 2 A, / i) . 
Now we define the following left-invariant, self-adjoint differential operator on N: 



U„ 



M* m M m 



(m)*^(m) 



Note that 



As aj m) G V 



ker U„ 



P| ker A 

3=1 



(m) 
'J 



t>), the operators and C/ m are homogeneous of degree m and 2m, 



respectively, w.r. to the dilation (29). Furthermore as M m is if- invariant, U m is also K- 
invariant. Hence it can be written as U m = u m (T>) where u m G V(M 
degree 2m with respect to the dilations (30) of M. d ". 



is homogeneous of 



By (28) 



Let 

Then 
(31) 



d-n\(U m )\ v ^ L ) = w m (£'(A,/i))Id . 
S m = {£' G E 6 , u m (0 = 0} . 

ker U m n S(N) K = {/ : supp 0/ C S m } . 



Moreover, 5 m is invariant under the dilations (30). 



The next lemma shows that polynomials which vanish on S m can be divided by u m . 
Lemma 5.1. Assume that p G "P(lR do ) vanishes on S m . Then p is divisible by u m . 
Proof. We may assume that p is homogeneous with respect to the dilations (30) of lR d °. 



Consider first the pairs in the first block of Table [2j 

In this case there is only one invariant in p D , leading to the sublaplacian on N, and then 
only one coordinate besides those corresponding to the T's. The space V{ji) coincides 



with P s '°(d) and by Proposition 4.6 V sfi (t>) C P s,0 (t)) <g> V m if and only if s > m. By 



Proposition 4.5, dir(M m ) vanishes on V s, °, if s < m. This is also the case for dit(U m ) = 
d-n-(M m )*d7r(M m ). Hence the set S m contains all the points of the form (|A|(2s + k), A) for 
any A G R dimi and s = 0, . . . , m - 1. 

We decompose p into its odd and even part w.r. to as 

^,A)=^Pi(^ 2 ,A)+p 2 (^ 2 ,A) , 

where p\ and P2 are two polynomials with suitable homogeneity. 

We claim that p\ and P2 must both vanish on the set of points (|A| 2 (2s + k) 2 , A) with 
A G ]R dim ^ and s = 0, . . . , m — 1. If it were not so, we would have the identity 



|A|(2s + k) 



P 2 (\\\ 2 (2s + k)\\) 
Pi (|A| 2 (2 S + k)2,A) ' 
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This contrasts with the fact that the right-hand side is a rational function in A, while the 
left-hand side is not. Then p 1 (j], A) andp 2 (?7,A) are both divisible by YlT^oiv - (2s + k;) 2 |A| 2 ). 
Therefore p(£[, A) is divisible by Yl™S (£'i 2 ~ (2s + k) 2 |A| 2 ). This also holds for p = u m . Hence 

771— 1 

(32) Mm (£,A)=cn(^ 2 -(2 S + «:) 2 |A| 2 ) . 

We consider next the pairs in the second block of Table [2j 

There are two invariants in p v for the pairs at lines 8 and 10 and three for the pair at line 
9. In the notation of Subsection 4.2.3 , the space V{p) coincides with V a ,i or V s ^j respectively, 



always with i and s of the same parity. We adopt the notation 

V(A,M) (lines 8,10) 
?(\,s,i,j) (line 9). 



e'(A,/i) 



More precisely, = |A|(2s + k) only depends on A and s. For the pair at line 9, £ 2 on ly 
depends on A, s,i, because it is invariant under the larger group U 2 x SU 2n . 



The homogeneity degrees of the elements of D w.r. to the dilations (29) are (1,2,1) at 



S; 



lines 8 and 10, and (1, 2, 2, 1) for the pair at line 9. By (30) and the subsequent comments, 
(33) £(M = |A|£(M) , m,s,t) = \%(l,s, l ) , £ 3 (\,s,i,j) = \%(1,s,i,j) . 
We split T,q as the union of = {£' : g do = 0} = p„(t)) x {0}, cf. [2], and the sets 

{£'(A, s, i) , AGR, s G z + 2N} , (lines 8, 10) 

{£'(\,s,i,j) , A G M , s ei + 2N , < j < (s -i)/2} , (line 9) 

depending on % > 0. 

By Propositions 4.7, 4.8, and 4.9, R s ^ (resp. R s ,i,j) is contained in (resp. V Sji j <g> 

V m ) if and only if z > m. By Proposition |4.5 and Proposition 4.10 dn(M m ) vanishes on V(p) 
if and only if R(p) is not included in V(p) <S> V m , which means i < m. This is also the case 
for dn(U m ) = dn (M m )* dn (M m ) . 

Hence S m contains the union of sets Si for < i < m — 1. Moreover, each polynomial u m 
vanishes on Si, i < m, but is never zero on Si, i > m, except for the origin. 

We prove recursively the existence of polynomials u, G "P(lR do ), i > 0, such that 

(a) u*(£,&A) = Cl ,^ 2 + £ 2 + d,A 2 , resp. (for line 9), fii = c M £ 2 + £ 2 + 
c 3,i^3 + ^*A ; 

(b) each Ui vanishes on Si but does not vanish on any other Si>, i' ^ i, except for the 
origin; 



scalar multiple of Ui - 



Once this is done, the proof can be concluded as in the previous case. 

Consider the polynomial u±. Being homogeneous of degree 2, it must be of the form 



(34) 

resp. 

(35) 



& A) = a^ 2 + a 2 £ 2 + 6A 2 + c£[\ 



& & A) = ai£'i 2 + a 2 £ 2 + a£ 3 + b\ 2 + c£A . 
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For i = 0, we have 

ai A 2 £ a (l, s) + a 2 A 2 £ 2 (l, 8, 0) +a 3 A 2 ^(l, s, 0, j) +b\ 2 + cA|A|£(l, s) = , 

v v ' 

only for line 9 

for every A 7^ 0, s even (and j < s/2). This forces c = by parity in A. 

In any case, we must have a 2 7^ 0. Suppose in fact that a 2 = 0. In the cases of lines 8, 10, 
the identity above would hold for every i, and u\ would vanish on every S{. In the case of line 
9, U\ would not depend on £ 2 and the polynomial ^(61,63) = ^1(61,63, 1) = ai£[ 2 + 03^3 + b 
would vanish at all points (2s + k, £ 3 (1, s, 0, j)) , for s even and j < s/2. Notice that, for s 
and i fixed, the values £ 3 (1, s, i,j) must all be different, because £ 3 is the only coordinate on 
Ejj depending on j. Then we would have p = and, by homogeneity, U\ = 0. 

Thus, we have obtained u\ = ui/a 2 satisfying (a), (b), (c) above. 

Assume now that we have constructed U{ G V{R d °), i = 0, . . . , i — 1, satisfying (a), (b), 
(c) above. Consider the polynomial Ui . It vanishes on Si, i < io, but does not vanish on Si, 
i > iq. Hence we can factor out Uj, i — 0, . . . , «q — 1, from Wj and there exists a polynomial 
qi such that Ui = qi ni=o Necessarily q it0 is homogeneous of degree 2 with respect to 
(30), and vanishes on Si because the polynomials Ui, i < io, do not vanish on it. Hence the 
quotient qi will have the form (34), resp. (35). Arguing as before, it can be shown that 
c = and a 2 ^ 0. Then u io = qi /a 2 has the required properties. □ 

The higher complexity of the second part of the proof given above was due to the presence 
of more than one polynomial in p , but also by the fact that we did not use explicit formulas 
for £' 2 (1, s,i) and £ 3 (1, s, i, j). To find such formulas does not seem an easy task anyhow, 
cf. [I]. On the other hand, the arguments used in the proof emphasise a pattern which is 
common to all cases at hand. 

Note that we have also proved the following identities: 

Ur 1 {(|A|(2s + k), A), A G M dim ^} (lines 8, 10) , 



^ (line 9) . 



Also note that what prevents S m from being an algebraic set is the dependence on |A| of 
£[. It follows from (32) and (33) that the zero set of u m in R d is S m U S 1 ", where 

S-m = {(~ £1)62,63, A) : (61,62,63, A) G S m ] , 

(with the £ 3 -component omitted for the pairs at lines 8, 10 - this caveat will not be repeated 
in the sequel). 

Let now G be a V m -valued, i^-equivariant Schwartz function G on N. Set / = M^G. 
Then / G S(N) K and / belongs to the orthogonal complement of f^^ker = kerU m . 
Hence the Gelfand transform of / vanishes on S m . The following lemma justifies that we 
can choose Schwartz extensions of Qf which vanish on S m . 

Proposition 5.2. Let f G S(N) K be such that its spherical transform Qf vanishes on S m . 
For any p G N, there exists ip = V 1 e *S(lR do ) such that: 

(i) (w m ^)|s 6 = Qf 

(ii) there exist C — C p > and q = q{p) such that H^Hfp) < C||/||(g)- 
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We state first a preliminary lemma. 

Lemma 5.3. Let P(y) be a real polynomial in y G W 1 . If f(x,y) G «S(R x M. n ) vanishes on 
{(P(y),y) : y G MJ 1 }, then there exists f G 5(1^°) satisfying f(x,y) — (x — P(y))f(x,y). 
Furthermore f depends linearly and continuously on f . 



Proof. The conclusion follows easily from Hadamard's lemma (Lemma 2.2), once we know 
that the change of variables (x, y) i — > (x — P(y),y) preserves 5(lR n+1 ) with its topology. 
This is trivial if degP(y) < 1. If degP = m > 1, it follows from the inequality 

\x-P(y)\ + \y\>C(\x\^ m +\y\) , 

which can be verified distinguishing between the two P(y)\ < \y\ and \x — P(y)\ > 

\y\- ' ' □ 

Proof of Proposition 5.2. Let tp G 5(lR a!o ) be an extension of Of. Such an extension exists 
by [2]. Let Pk be the homogeneous component of degree k with respect to (30) in the Taylor 
expansion of ip around the origin. Since p vanishes on S m , which is invariant under these 
dilations, Pk vanis hes on S m . 

By Lemma 5.1 there exists Qk G ^(IR^ ) homogeneous of degree k with respect to (30) 
such that u m Q k = P k +2m- 

Applying Whitney's extension theorem, there exists ip\ G (^^(IR^ ) with compact support 
around the origin and Taylor expansion ^ fceN Qk at the origin. Then tp — u m ip\ vanishes of 
infinite order at the origin. 

We take now a function rj, homogeneous of degree w.r. to the dilations (30), C°° 
away from the origin, and equal to 1 on a conic neighbourhood of and equal to on 
a conic neighbourhood of S 1 ". Such a function exists because, by the hypoellipticity of the 
sublaplacian, is contained in a conic region around the positive ^-semiaxis, cf. e.g. (15) 
in 0: 

£dC{(£UU3,A):|# + |# + |A|<C<£} . 

Then the function co = (ip — u m ipi)rj is Schwartz and vanishes on S m U S 1 ". By repeated 
application of Lemma 5.3[ u = u m ip2, with ip2 Schwartz. Take ip = ipi+ip2- Then u m ipr] = prj, 
so that (i) holds. 

Consider now the Schwartz norm H^llfr) < HV'iIIcp) + 1 1^2 1 1 

By Lemma 5.3, there exist an integer v = u(p) > p and a constant A p such that 

\\lp2\\(p) < A p \\u\\ (u) < A' p \\<p - UmlpxW^) < Ap(\\ip\\ {v) + \\lpi\\(u+2m)) ■ 

In order to estimate ||V ; i||(i/+2m), we use the fact that the Whitney extension of the jet 
{QfcjfceN can be performed so that the resulting function ipi = xp^ satisfies, for an integer 
r = r(p) and a constant B p , 

-Pfc+2m|| < -B p ||V 9 ll(r+2m) > 

k<r k<r 

where the norm of a polynomial is meant as the maximum of its coefficients. 
Putting all together, 

^ Cpllvll (max(r,i/)+2m) • 
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By [2], there are an integer q = q(p) and a constant C p such that it is possible to choose 
cp = pfr) above so that 

|MI(max(r,v)+2m) < Cp||/||(g) , 

and this concludes the proof. □ 
We resume the proof of Proposition |4.3[ 



Given G, set / = M^G G S(N) K G (kerC/J 1 . By ((3Xj , £/ vanishes on S m . 
Applying Proposition 5.2, we can choose a Schwartz function ^ such that u m ip extends 
Qf. Defining h = 1 (V>)> we easily obtain, on Eg, 

Q(U m h) = u m ip = Qf ■ 

This implies 

M* M m /* = u m h = f= \rj: . 

To factor out M^, observe that for any A ^ 0, 

cMM m )*7r A (M m /i - H) = . 



By Proposition 4.5 both sides are when restricted to a subspaces V(/j) with /i non-m- 
admissible. If \i is m-admissible, then Proposition 4.10 implies that tt\ (M m h — H) = on 
V(fj). Then M m /i = H. 

It remain s to prove the estimates on the Schwartz norms. To the norm estimates given by 
Proposition 5.2 it is sufficient to add that and Q~ x are continuous on the appropriate 



Schwartz spaces. For ^ 1 we refer to [2J [3, IE]- 



6. Conclusion 



We complete the proof of Theorem 1.1 



Let G G (S(N) <g> P fc (3o)) as in Q. We decompose G as in g. We realise the 
representation space V m as W a when |a| = m. By Lemma 4.3, for each (a, 0), [a] + [(3} = k, 
there exists h a ^ G S(N) K such that 

P a (v,()g a i3 = M at( h a ,j3 , 

where the operator M a ^ = Ylj=i ^j"^ o ; (0 * s ^ ne rean sation °f on W a . 

In the notation of ([5]), the operators Z?"" form a basis of (O(iV) £g> V($o)) K ■ Therefore, 
each M a ^ can be expressed as a linear combination of the Df' with [a"j = k, and one can 
write G as 



G 



£ 6 



Hr.ii 



where the functions if n 



[a"]=k 

are finite linear combinations of h 



The norm estimates are obvious after Proposition 4.3 
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